Remote surgical mentoring

ABSTRACT

A virtual representation of an operating room is generated based on robot information and sensing of the OR with depth cameras. One of the depth cameras is integrated with a portable electronic device, operated by a local user in the operating room. The virtual representation of the OR is communicated to the virtual reality headset, with three-dimensional point cloud data. A virtual reality environment is rendered to a display of the virtual reality headset, operated by a remote user. A virtual representation of the remote user is rendered in augmented reality to a display of the portable electronic device.

TECHNICAL FIELD

This disclosure relates generally to the field of surgical robotics and,more particularly, to remote surgical mentoring.

BACKGROUND

Minimally-invasive surgery (MIS), such as laparoscopic surgery, involvestechniques intended to reduce tissue damage during a surgical procedure.For example, laparoscopic procedures typically involve creating a numberof small incisions in the patient (e.g., in the abdomen), andintroducing one or more tools and at least one endoscopic camera throughthe incisions into the patient. The surgical procedures are thenperformed by using the introduced tools, with the visualization aidprovided by the camera.

Generally, MIS provides multiple benefits, such as reduced patientscarring, less patient pain, shorter patient recovery periods, and lowermedical treatment costs associated with patient recovery. In someembodiments, MIS may be performed with surgical robotic systems thatinclude one or more robotic arms for manipulating surgical instrumentsbased on commands from an operator. For example, an operator may providecommands for manipulating surgical instruments, while viewing an imagethat is provided by a camera and displayed on a display to the user.

Performance of a surgical procedure can require in-depth knowledge andexperience regarding different aspects of the surgery, including setup,workflow, and more. The use of surgical robotic systems potentiallyincreases the complexity of the surgery. It can be helpful for a surgeonor assistant to collaborate pre-operatively, intra-operatively, and postoperatively, with remotely located medical professionals. Procedures canbe improved upon, and risk can be identified and reduced, through remotecollaboration.

SUMMARY

Generally, a surgical robotic system has remote mentoring features thatallow for pre-operative, intraoperative, and post-operativecollaboration. Remote mentoring features can provide an intuitive androbust method for guiding intraoperative procedures via a remote mentor.A remote user (a ‘mentor’) wears a virtual reality (VR) headset toorchestrate an avatar, which is a digital representation of the mentor,that is virtually introduced into the operating room and visible by alocal user that is controlling or wearing an augmented reality (AR)device such as a computer tablet, mobile smartphone, or headset (e.g.,augmented reality or mixed reality glasses). AR refers to superimposingvirtual objects (e.g., a digital representation of the mentor) over avideo feed showing the physical environment of the AR user, in thiscase, the operating room. The virtual objects are typically integratedinto the physical environment of the user, for example, by beinginteractive with the physical environment. It should be understood that,for the purpose of this disclosure, the terms augmented reality andmixed reality are interchangeable.

The remote VR user and the local intraoperative AR user are both boundedin a shared coordinate frame relative to the Verb robotic system. Inother words, they have a common coordinate system so that they areco-located in the same virtual space in a realistic manner. The remotevirtual reality user is visible not only through the avatar, but canalso create notes and markings that are localized within 3D space,provide recommendations and examples for robotic arm movement/positions,and recommend surgical table orientation/position. Audio streams of theremote and local user can be transmitted back and forth between theremote and local users, to facilitate verbal communication. Thus, inputs(e.g., the notes, markings, recommendations, and/or audio) from theremote user can provide the local user with helpful insight regardingdifferent aspects of a surgical procedure.

In turn, the point cloud information of the physical operating roomenvironment can be captured by depth cameras and transmitted back to thevirtual reality environment for the remote user to view and manipulate.By using an augmented reality headset/tablet as a sensor source and datagenerator, the remote VR user can construct a 3D mesh of the patient inVR that will instruct and enable intraoperative guidance.

In some embodiments, the system can include a plurality of depth cameras(e.g., RGBD scanners or equivalent technology). At least one of thedepth cameras is integrated with a portable electronic device operatedby a local user in an operating room (OR). Another one of the depthcameras is arranged in the OR, such as being mounted on equipment and/orwalls. The system can include a surgical robot, such as robot arms andplatform for supporting a patient during surgery, and controller forcommanding movements of the surgical robot.

A virtual representation of the OR is generated based on a) robotinformation received from the surgical robot or controller, and b)sensing of the OR by the plurality of depth cameras. The virtualrepresentation of the OR and three dimensional point cloud data istransmitted to a virtual reality headset used to render a virtualreality environment to a display of the virtual reality headset worn bya remote user. In this manner, a remote user can ‘see’ what is happeningin the OR and provide insight.

A virtual representation of the remote user is rendered in augmentedreality at the portable electronic device. The virtual representationcan be a graphical object superimposed over a ‘live’ video feed of theOR. The virtual representation of the remote user can be generated basedon position of the remote user, such that the virtual representation ofthe remote user shares a common coordinate system that is common to thevirtual representation of the OR, the surgical robot, and the portableelectronic device of the local user. In other words, if the remote usermoves forward towards the patient in the virtual representation of theOR (as shown to the remote user in the virtual reality headset), theavatar of the remote user will simultaneously move forward towards thepatient as shown in the augmented reality displayed to the local user.Similarly, if the remote user ‘points’ at a section of the patient inthe virtual reality environment that is shown to the remote user, theavatar of the remote user will point at the corresponding section of thepatient in the augmented reality shown to the local user. This creates a‘realistic’ interaction between the local and remote user and merges theaugmented reality of the local user with the virtual reality of theremote user.

Inputs from the remote user such as audio data (e.g., the remote user'sspeech) and virtual annotations generated by a digital tool (e.g.,handheld user input device held by the remote user) can be indicatedthrough the virtual representation of the remote user. The virtualrepresentation can be a simplified rendering of a person (e.g., a headand hands), a realistic representation, or a symbolic representation,that communicates the remote user's gestures, movements, position,and/or intention to the user.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows an example of a surgical robotic system in an operatingroom according to some embodiments.

FIG. 2 shows a surgical robotic system with remote telementoringfeatures according to some embodiments.

FIG. 3 illustrates a remote user being virtually imported into anoperating room in an augmented reality experience, according to someembodiments.

FIG. 4 illustrates an operating room and local user being virtuallyimported into a virtual reality experience of a remote user, accordingto some embodiments.

FIG. 5 shows a method for remote mentoring with a surgical roboticsystem, according to some embodiments.

DETAILED DESCRIPTION

Non-limiting examples of various aspects and variations of the inventionare described herein and illustrated in the accompanying drawings.

Referring to FIG. 1, this is a pictorial view of an example surgicalrobotic system 1 in an operating arena. The system 1 includes a userconsole 2, a control tower 3, and one or more surgical robotic arms 4 ata surgical robotic platform 5, e.g., a table, a bed, etc. The arms 4 maybe mounted to a table or bed on which the patient rests as shown in theexample of FIG. 1, or they may be mounted to a cart separate from thetable or bed. The system 1 can incorporate any number of devices, tools,or accessories used to perform surgery on a patient 6. For example, thesystem 1 may include one or more surgical tools 7 used to performsurgery. A surgical tool 7 may be an end effector that is attached to adistal end of a surgical arm 4, for executing a surgical procedure.

Each surgical tool 7 may be manipulated manually, robotically, or both,during the surgery. For example, the surgical tool 7 may be a tool usedto enter, view, or manipulate an internal anatomy of the patient 6. Inone aspect, the surgical tool 7 is a grasper that can grasp tissue ofthe patient. The surgical tool 7 may be configured to be controlledmanually by a bedside operator 8, robotically via actuated movement ofthe surgical robotic arm 4 to which it is attached, or both. The roboticarms 4 are shown as being table-mounted but in other configurations thearms 4 may be mounted to a cart, the ceiling or a sidewall, or toanother suitable structural support.

A remote operator 9, such as a surgeon or other human operator, may usethe user console 2 to remotely manipulate the arms 4 and their attachedsurgical tools 7, e.g., referred to here as teleoperation. The userconsole 2 may be located in the same operating room as the rest of thesystem 1 as shown in FIG. 1. In other environments however, the userconsole 2 may be located in an adjacent or nearby room, or it may be ata remote location, e.g., in a different building, city, or country. Theuser console 2 may comprise a seat 10, foot-operated controls 13, one ormore handheld user input devices, UID 14, and at least one user display15 that is configured to display, for example, a view of the surgicalsite inside the patient 6. In the example user console 2, the remoteoperator 9 is sitting in the seat 10 and viewing the user display 15while manipulating a foot-operated control 13 and a handheld UID 14 inorder to remotely control the arms 4 and the surgical tools 7 that aremounted on the distal ends of the arms 4.

In some variations, the bedside operator 8 may operate the system 1 inan “over the bed” mode in which the beside operator 8 (user) is at aside of the patient 6 and is simultaneously manipulating arobotically-driven tool (an end effector that is attached to the arm 4)with a handheld UID 14 held in one hand, and a manual laparoscopic toolin another hand. For example, the bedside operator's left hand may bemanipulating the handheld UID to control a robotically-driven tool,while the bedside operator's right hand may be manipulating a manuallaparoscopic tool. In this particular variation of the system 1, thebedside operator 8 can perform both robotic-assisted minimally invasivesurgery and manual laparoscopic surgery on the patient 6.

During an example procedure (surgery), the patient 6 is prepped anddraped in a sterile fashion to achieve anesthesia. Initial access to thesurgical site may be performed manually while the arms of the roboticsystem 1 are in a stowed configuration or withdrawn configuration (tofacilitate access to the surgical site.) Once access is completed,initial positioning or preparation of the robotic system 1 including itsarms 4 may be performed. Next, the surgery proceeds with the remoteoperator 9 at the user console 2 utilizing the foot-operated controls 13and the UIDs 14 to manipulate the various end effectors and perhaps animaging system, to perform the surgery. Manual assistance may also beprovided at the procedure bed or table, by sterile-gowned bedsidepersonnel, e.g., the bedside operator 8 who may perform tasks such asretracting tissues, performing manual repositioning, and tool exchangeupon one or more of the robotic arms 4. Non-sterile personnel may alsobe present to assist the remote operator 9 at the user console 2. Whenthe procedure or surgery is completed, the system 1 and the user console2 may be configured or set in a state to facilitate post-operativeprocedures such as cleaning or sterilization and healthcare record entryor printout via the user console 2.

In one embodiment, the remote operator 9 holds and moves the UID 14 toprovide an input command to move a robot arm actuator 17 in the roboticsystem 1. The UID 14 may be communicatively coupled to the rest of therobotic system 1, e.g., via a console computer system 16. The UID 14 cangenerate spatial state signals corresponding to movement of the UID 14,e.g. position and orientation of the handheld housing of the UID, andthe spatial state signals may be input signals to control a motion ofthe robot arm actuator 17. The robotic system 1 may use control signalsderived from the spatial state signals, to control proportional motionof the actuator 17. In one embodiment, a console processor of theconsole computer system 16 receives the spatial state signals andgenerates the corresponding control signals. Based on these controlsignals, which control how the actuator 17 is energized to move asegment or link of the arm 4, the movement of a corresponding surgicaltool that is attached to the arm may mimic the movement of the UID 14.Similarly, interaction between the remote operator 9 and the UID 14 cangenerate for example a grip control signal that causes a jaw of agrasper of the surgical tool 7 to close and grip the tissue of patient6.

The surgical robotic system 1 may include several UIDs 14, whererespective control signals are generated for each UID that control theactuators and the surgical tool (end effector) of a respective arm 4.For example, the remote operator 9 may move a first UID 14 to controlthe motion of an actuator 17 that is in a left robotic arm, where theactuator responds by moving linkages, gears, etc., in that arm 4.Similarly, movement of a second UID 14 by the remote operator 9 controlsthe motion of another actuator 17, which in turn moves other linkages,gears, etc., of the robotic system 1. The robotic system 1 may include aright arm 4 that is secured to the bed or table to the right side of thepatient, and a left arm 4 that is at the left side of the patient. Anactuator 17 may include one or more motors that are controlled so thatthey drive the rotation of a joint of the arm 4, to for example change,relative to the patient, an orientation of an endoscope or a grasper ofthe surgical tool 7 that is attached to that arm. Motion of severalactuators 17 in the same arm 4 can be controlled by the spatial statesignals generated from a particular UID 14. The UIDs 14 can also controlmotion of respective surgical tool graspers. For example, each UID 14can generate a respective grip signal to control motion of an actuator,e.g., a linear actuator, which opens or closes jaws of the grasper at adistal end of surgical tool 7 to grip tissue within patient 6.

In some aspects, the communication between the platform 5 and the userconsole 2 may be through a control tower 3, which may translate usercommands that are received from the user console 2 (and moreparticularly from the console computer system 16) into robotic controlcommands that transmitted to the arms 4 on the robotic platform 5. Thecontrol tower 3 may also transmit status and feedback from the platform5 back to the user console 2. The communication connections between therobotic platform 5, the user console 2, and the control tower 3 may bevia wired and/or wireless links, using any suitable ones of a variety ofdata communication protocols. Any wired connections may be optionallybuilt into the floor and/or walls or ceiling of the operating room. Therobotic system 1 may provide video output to one or more displays,including displays within the operating room as well as remote displaysthat are accessible via the Internet or other networks. The video output(video feed) may also be encrypted to ensure privacy and all or portionsof the video output may be saved to a server or electronic healthcarerecord system.

FIG. 2 shows the surgical robotic system 1 having remote features thatallow surgeons to collaborate pre-operatively, intra-operatively, andpost-operatively. The development of a real-time virtual reality andaugmented reality collaboration platform enables an interaction paradigmin which a remote surgical professional can conveniently observe andadvise surgeries from disparate locations.

A processor 24 can generate a virtual representation of an operatingroom based on a) robot information, and b) sensing of the operating roomby one or more depth cameras (e.g., cameras 21, 23). The depth camerascan include at least one depth camera 23 integrated with a portableelectronic device 22 operated by a local user in the operating room, andat least one stationary depth camera 21 arranged in the operating roomat a fixed location (e.g., on walls and/or on equipment). The depthcameras can be RGBD sensors, or other equivalent technology that sensescolor and depth.

The robot information is generated by components of the surgical roboticsystem shown in FIG. 1. For example, the surgical robotic platform canprovide telemetry generated by servos or sensors that indicates positionand orientation (e.g., platform height and angle) at which the platformis currently held at. Similarly, sensors or controllers of the surgicalrobot arms and surgical tool can provide telemetry that describes theposition (e.g., joint angles) of the surgical robotic arms, tool typesthat are currently attached to the robotic arms, tool grasp status(e.g., 90% closed), and active tool energy. This data can be collectedat the control tower or user console. Further, system information suchas system state (on, off, idle) or error code can be collected andshared with a remote user.

The processor can use the robot information to generate a virtualrepresentation of the respective robotic components in the positionsindicated by the robot information based on, for example, the roboticjoint angles and the platform height and angle. This can obviate theneed to extrapolate precise positioning of the platform, robotic armsand tools from image data sensed by the depth cameras, while alsoimproving accuracy of the virtual representation of these components,which is included in the virtual representation of the operating room.

In some embodiments, the portable electronic device 22 can be a tabletcomputer or mobile smartphone 28. Augmented reality (e.g., therepresentation of remote user and/or other virtual objects) is renderedon the display of the device, superimposed over a stream of imagescaptured by the device. Alternatively, the portable electronic devicecan be a head worn device such as augmented reality or mixed realityglasses 39, having a transparent display worn in front of eyes, ontowhich the augmented reality is rendered over. This is done in‘real-time’ or ‘live’, meaning that the processing and display of theaugmented reality is performed contemporaneous with the capturing of thestream of images, notwithstanding unavoidable delays such as buffering,processing, and communication latency.

A virtual representation of the operating room and three dimensionalpoint cloud data is transmitted to a virtual reality headset 26(operated by a remote user) used to render a virtual reality environmentto a display of the virtual reality headset. The virtual realityenvironment here is fully immersive, meaning that the content shown tothe remote user is not integrated with the remote user's environment,rather the remote user is virtually transported to the operating room.

The virtual representation of the operating room can include athree-dimensional rendering of the operating room and components in theoperating room. In some cases, the virtualized operating room caninclude metadata such as equipment type, location and orientation, sothat the virtual reality headset can render those objects into thevirtual reality environment, thus reducing the amount of data that mustbe transmitted to the virtual reality headset, for example, if raw 3Dpoint cloud data for all objects of the environment was to betransmitted. The virtual representation of the operating room caninclude mesh representations of objects, walls, ceiling, personnel(e.g., surgeons, assistants, etc.) that are sensed in the ORenvironment. The virtual representation of the operating room can beconstructed based on 3D point cloud data that is captured by theplurality of depth cameras.

The virtual representation can be generated by processor 24. Theprocessor can be integral to a standalone computer, integrated with thecontroller, a laptop, the portable electronic device that is operated bythe local user, or other computing systems of the surgical roboticsystem.

It is recognized, however, that for some areas, it is beneficial totransmit raw 3D point cloud data to the remote user, to provide greaterdetail and resolution, and to allow the remote user greater ability tointeract with the sensed environment of the operating room. This 3Dpoint cloud data can be generated by the depth camera of the portableelectronic device 22, or by one or more depth cameras mounted in the ORon equipment or walls, directed at an area of interest. For example, thelocal user can aim the depth camera 23 of the portable electronic deviceat areas of interest on a patient (e.g., at a patient abdomen). This raw3D point cloud data can be transmitted to the remote VR headset to allowthe remote user to analyze and manipulate this raw image data.

For example, remote surgeons can perform segmentation of organs on apatient as another surgeon is performing a surgery so thatintraoperative personnel can receive real-time visual guidance onphysiological landmarks. Segmentation refers to the process of dividingan image into regions with similar properties such as gray level, color,texture, brightness, and contrast. Thus, the remote user can subdivideobjects in the image to help identify regions of interest (e.g., partsof organs, tumors, lesions, other abnormalities). Based on the raw pointcloud data, the remote user can also identify locations on a patient forports that serve as entry points for the surgical robotic instruments.Thus, having a depth camera that can be directed to areas of interestallows the local user to send detailed scans of the patient to theremote user.

The processor uses the robot information and the sensor data sensed bythe depth cameras to recognize and virtually establish the 3D positionand orientation of the surgical robotic system and its components in theAR device's coordinate frame. This can be performed through knowncomputer vision algorithms, such as image classification and objectlocalization, and performing transformations from one coordinate systemto another to determine a common coordinate system.

Through this process, the depth cameras arranged across surgical roboticsystem and the portable electronic device establish a common coordinateframe and create a virtual recreation of the robotic arms/table thatmatches the real-time position/orientation of the robotic arms/table inthe room. After the components of the surgical robotic system arelocalized in 3D space (e.g. the control tower, surgeon bridge, and tableand arms), a remote participant can interact with a virtualreconstruction of the OR. The virtual representation of the remote user(e.g., orientation/position of the remote user's hands, body, and/orhead) is communicated to the augmented reality user in the same relativecoordinate frame.

A simultaneous localization and mapping (SLAM) algorithm is performedupon data generated by the at least one depth camera integrated with theportable electronic device, to localize the position and orientation ofthe portable electronic device 3D space and in the common coordinatesystem.

The portable electronic device of the local user and/or the processor 24can have a data connection to the surgical robotic system so that it canretrieve real-time robot information (e.g., the robot's arm jointangles, tool types, tool grasp status, active tool energy, arm modes,table position and orientation, the case surgeon, and troubleshootingerror codes). This can be used to generate the virtual OR, and providenotifications to the local user and remote user.

In addition to localizing the components (e.g., the platform, userconsole, control tower, robotic arms) of the surgical robotic system in3D space, the remote user is sent a live “window” into the 3Dintraoperative space through a 3D point-cloud stream that is generatedby the one or more depth cameras (e.g., 23 and/or 21). This point cloudprovides a real-time stream of actual depth and RGB data from the OR,rather than a simplified stream of component position/orientation datathat adjusts pre-rendered components of the surgical robotic system andOR components in the remote user's virtual reality environment. Thispoint cloud stream, displayed in the correct coordinate space relativeto the rest of the virtual reality environment, unlocks additionalcapabilities of the remote user.

For example, the remote user can see the patient's body in greaterdetail through the 3D point cloud data (or a mesh generated based on the3D point cloud data), and create digital waypoints and virtual guidesfor where trocars should be placed in a patient (see, for example, FIG.4). Additionally, or alternatively, the remote user can create digitalwaypoints and virtual guides for how the robotic arms should be orientedrelative to the patient's body. Additionally, or alternatively, theremote user can give educated advice about where staff should bestanding in the operating room, or where equipment should be arranged inthe operating room. The local user interacts with the remote userthrough the portable device 22, as shown in FIG. 3.

FIG. 3 illustrates the operating room and a virtual representation ofthe remote user placed in the operating room, creating an augmentedreality experience for the local user. A virtual representation of theremote user is rendered in augmented reality to a display of theportable electronic device 22 based on position of and input from, theremote user. The position of the remote user shares a common coordinatesystem, i.e. it is common to the virtual representation of the OR, andthe portable electronic device of the local user.

The position of the remote user, relative to the OR, can be determinedusing one or more points of reference at the user's remote location. Forexample, at the remote location of the remote user, sensors can bearranged in the surroundings of the remote user, and/or on the VRheadset. A point of origin can be selected (e.g., arbitrarily) in theremote user's location to match a point of origin in the space of theoperating room. Transformations can be performed to map movement andlocation of the remote user to the virtual representation of the OR.Thus, location and orientation of the remote user can be calculated inrelation to the coordinate system of the operating room, allowing theremote user to virtually explore and interact with the operating room.

As discussed in other sections, the local user can aim a depth camera,which can be integrated with the portable device 22, at a location ofinterest, such as the patient's abdomen. Raw point cloud data generatedby the depth camera can be transmitted in real-time to the remote user.Virtual markings and objects can be rendered over the display of theportable device, based on input generated by the remote user, such as:virtual annotations generated by a digital tool remotely, anatomicalsegmentation data, indications of trocar placement, and previoussurgical set up.

For example, a remote user can use a handheld digital tool 27 as shownin FIG. 2 to circle areas to cut or remove, to mark where a port ofentry for a trocar should be, or to mark where a suture is needed. These‘marks’ can be virtually represented by virtual objects (e.g., lines,highlighted areas, or shapes) in the local user's portable device.

As discussed, input from the remote user can include audio data capturedat the VR headset. Audio from the local user can simultaneously betransmitted to the VR headset, providing a two-way communication. Voiceover IP (VOIP) or other audio communication technology can be used.

Referring back to FIG. 3, the local user can ‘see’ and ‘hear’ the remoteuser as the remote user verbally instructs and marks a location forwhere a trocar should be inserted. The virtual presence of the remoteuser creates a natural telementoring platform. Spatially contextual 3Dmarkings and notes are communicated from the remote user to the localuser. This can be contemporaneous with performance of a surgery. Theremote user can provide visual guidance not only in the endoscope feed,but also around the robot, for example, how to orient arms, where toplace endoscope and tools, where a 1st assistant should stand. Thevirtual representation of the remote user can include one or more of thefollowing: a) a virtual representation of the virtual reality headset,b) virtual hands indicating hand movement or hand location of the remoteuser, and c) a virtual avatar resembling part of or a whole human.

FIG. 4 shows a virtual reality experience of the remote user. The remoteuser is transported to the operating room where the surgical roboticsystem is located. The remote user, wearing the VR headset 26, canexplore the virtual space and provide inputs verbally and/or using aremote digital tool such as a digital pen or a handheld UID (e.g.,remote digital tool 27 shown in FIG. 2).

The virtualized OR data and the point cloud data is communicated to theVR headset where it is used to render the virtual representation of theOR, and detailed areas of points of interest (e.g., the patient) onto adisplay of the remote user's VR headset. In addition, the position andorientation of the portable electronic device 22 of the local user canbe communicated to the virtual reality headset. This can be used torender a virtual representation of the portable electronic device (e.g.,floating in the virtual environment) and/or a virtual representation ofthe local user (e.g., an avatar) in the virtual reality environment. Inthis manner, the remote user can feel like she is interacting with thelocal user in the virtual OR.

As discussed in other sections, the remote user can use medical imagingsegmentation to highlight, segment, and visually annotate tissues andorgans that are relevant to the operation as the operation isproceeding. The remote user can load and overlay their own, previoussurgical setups onto the augmented reality user's existingintraoperative setup as a guide for how they can organize their ownOR/robot.

A method 40 is shown in FIG. 5 that provides interactive and immersiveremote mentoring, according to some embodiments. The method can beperformed by the surgical robotic system shown in FIG. 1 and FIG. 2. Forexample, the method can be performed by processor 24, which can beintegral to a standalone computer, integrated with components of thesurgical robotic system 1 (shown in FIG. 1 or FIG. 2), or integratedwith the portable electronic device 22 of FIG. 2.

At operation 41, the method includes generating a virtual representationof an OR based on a) robot information, and b) sensing of the OR by oneor more depth cameras. The depth cameras include at least one depthcamera integrated with a portable electronic device operated by a localuser in the OR, and at least one stationary depth camera arranged in theOR. The robot information can be used to more accurately generatepositions of the robotic components such as the robot arms,tools/instruments attached thereto, and the surgical platform.

At operation 42, the method includes transmitting the virtualrepresentation of the OR and three dimensional point cloud data to avirtual reality headset where this is used to render a virtual realityenvironment to a display of the virtual reality headset, operated by aremote user. The virtual representation of the OR can include mesh datadescribing geometry of the operating room and objects in the room,and/or position and orientation data of recognized objects in theoperating room. The remote user is virtually transported into theoperating room, so that the remote user can analyze the setup, workflow,or provide other helpful insights to perform a surgery.

At operation 43, the method includes rendering a virtual representationof the remote user in augmented reality to a display of the portableelectronic device based on position of and input from, the remote user.The virtual representation of the remote user is rendered in a commoncoordinate system that is common to the virtual representation of theOR, and the portable electronic device of the local user. The commoncoordinate system can be determined based on transformations betweenimage data captured by a depth camera attached to or integrated with theportable electronic device, and image data captured by depth camerasmounted in the operating room.

The method can be performed continuously and in real-time. For example,operation 42 and operation 43 are performed continuously andsimultaneously to provide a cohesive interaction between the local userand the remote user.

The foregoing description, for purposes of explanation, used specificnomenclature to provide a thorough understanding of the invention.However, it will be apparent to one skilled in the art that specificdetails are not required in order to practice the invention. Thus, theforegoing descriptions of specific embodiments of the invention arepresented for purposes of illustration and description. They are notintended to be exhaustive or to limit the invention to the precise formsdisclosed; obviously, many modifications and variations are possible inview of the above teachings. The embodiments were chosen and describedin order to best explain the principles of the invention and itspractical applications, and they thereby enable others skilled in theart to best utilize the invention and various embodiments with variousmodifications as are suited to the particular use contemplated.

What is claimed is:
 1. A method for remote mentoring with a surgicalrobotic system, comprising: generating a virtual representation of anoperating room (OR) based on a) robot information, and b) sensing of theOR by one or more depth cameras, including at least one depth cameraintegrated with a portable electronic device operated by a local user inthe OR, and at least one stationary depth camera arranged in the OR;transmitting the virtual representation of the OR and three dimensionalpoint cloud data to a virtual reality headset used to render a virtualreality environment to a display of the virtual reality headset,operated by a remote user; and rendering a virtual representation of theremote user in augmented reality to a display of the portable electronicdevice based on position of and input from, the remote user.
 2. Themethod of claim 1, wherein the three dimensional point cloud dataincludes a patient in the OR.
 3. The method of claim 1, wherein theinput from the remote user includes audio data captured at the virtualreality headset.
 4. The method of claim 1, wherein the input from theremote user includes at least one of: virtual annotations generated by adigital tool remotely, anatomical segmentation data, trocar placement,and previous surgical set up, to be rendered in the augmented reality atthe portable electronic device.
 5. The method of claim 1, wherein theportable electronic device of the local user is a tablet computer andthe augmented reality is rendered on the display over a stream of imagescaptured by the tablet computer.
 6. The method of claim 1, wherein theportable electronic device of the local user is a head worn device witha transparent display worn in front of eyes, onto which the augmentedreality is rendered over.
 7. The method of claim 1, wherein the robotinformation includes at least one of: joint angles of one or moresurgical robotic arms, tool types, tool grasp status, active toolenergy, position and orientation of a surgical table, and an error codeof the surgical robotic system.
 8. The method of claim 1, whereinsimultaneous localization and mapping (SLAM) is performed upon datagenerated by the at least one depth camera integrated with the portableelectronic device, to localize the position and orientation of theportable electronic device in the common coordinate system.
 9. Themethod of claim 1, wherein the position and orientation of the portableelectronic device is communicated to the virtual reality headset, usedto render a virtual representation of the portable electronic device ora virtual representation of the local user in the virtual realityenvironment of the remote user.
 10. The method of claim 1, wherein theone or more depth cameras include RGBD cameras.
 11. The method of claim1, wherein the virtual representation of the remote user includes atleast one of: a) a virtual representation of the virtual realityheadset, b) virtual hands indicating hand movement or hand location ofthe remote user, or c) a virtual avatar resembling part of or a wholehuman.
 12. A surgical robotic system, comprising: a plurality of depthcameras, including at least one depth camera integrated with a portableelectronic device operated by a local user in an operating room (OR),and at least one stationary depth camera arranged in the OR; a surgicalrobot and controller; and a processor, configured to perform thefollowing: generating a virtual representation of the OR based on a)robot information received from the surgical robot or controller, and b)sensing of the OR by the plurality of depth cameras; transmitting thevirtual representation of the OR and three dimensional point cloud datato a virtual reality headset, causing the virtual reality headset torender a virtual reality environment to a display of the virtual realityheadset worn by a remote user; and transmitting, to the portableelectronic device of the local user, position of, and input from, theremote user, used to render a virtual representation of the remote userin augmented reality at the portable electronic device.
 13. The surgicalrobotic system of claim 12, wherein the three dimensional point clouddata includes a body of a patient.
 14. The surgical robotic system ofclaim 12, wherein the input from the remote user includes audio datacaptured at the virtual reality headset.
 15. The surgical robotic systemof claim 12, wherein the input from the remote user includes at leastone of: virtual annotations generated by a digital tool remotely,anatomical segmentation data, trocar placement, and previous surgicalset up, to be rendered in the augmented reality at the portableelectronic device.
 16. The surgical robotic system of claim 12, whereinthe portable electronic device of the local user is a tablet computerand the augmented reality is rendered over a stream of images capturedby the tablet computer and displayed on a screen of the tablet computer.17. The surgical robotic system of claim 12, wherein the portableelectronic device of the local user is a head worn display with atransparent display area onto which the augmented reality is renderedover.
 18. The surgical robotic system of claim 12, wherein the robotinformation includes at least one of: joint angles of one or moresurgical robotic arms, tool types, tool grasp status, active toolenergy, position and orientation of a surgical table, and an error codeof the surgical robotic system.
 19. The surgical robotic system of claim12, wherein simultaneous localization and mapping (SLAM) is performedupon data generated by the at least one depth camera integrated with theportable electronic device, to localize the position and orientation ofthe portable electronic device in the common coordinate system.
 20. Thesurgical robotic system of claim 12, wherein the position andorientation of the portable electronic device is communicated to thevirtual reality headset, used to render a virtual representation of theportable electronic device or a virtual representation of the local userin the virtual reality environment of the remote user.